tree: bound each reorganization job so a hung host call cannot block the queue by HugoXK · Pull Request #2758 · ThinkFlowLab/FlyBot

HugoXK · 2026-11-10T00:22:07Z

tree: bound each reorganization job so a hung host call cannot block the queue

Bot: Dev-5
Owner: Dev-5
Written-by: Dev-5
Fixes #2751

What was wrong

plugins/tree/src/organize.js keeps one serial reorganization queue (pending plus a promise chain) and had no bound on a job. One placement, repair or allocation whose host call (fb.bot.create / fb.bot.updateBot) never answered kept pending > 0 for ever: every later exclusive() job never ran and idle() (exported as settled) never resolved (#2751).

What changed

Every queued job now runs under a deadline: DEFAULT_JOB_TIMEOUT_MS (10 minutes), injectable as jobTimeoutMs for tests.
When the deadline passes the queue logs flybot-tree: reorganization job <id> was abandoned after <ms>ms: the host call never settled; later reorganizations continue, drops that job's result and lets the next job run, so a failed or abandoned job cannot poison the queue.
An exclusive() job reports the give-up instead of hanging: its promise rejects with an Error carrying abandoned: true and jobId, so a caller can tell a give-up from a failure.
An abandoned placement clears its own organizing flag (its job's finally never runs), so the next adoption pass can place that Bot instead of leaving it marked organizing for ever.
docs/subsystems/tree.md records the bound and changelog.d/2751-reorg-queue-bound.fixed.md is the user-visible note.

Red first

At base f2ef4dfaccedd40315bd585474d0902118dc0ba8, in plugins/tree:

$ node --import ./test/resolve.mjs --test --test-timeout=2500 test/organize-queue-bound.test.js
# tests 3 / # pass 0 / # fail 0 / # cancelled 3   (exit 1)

The first case is cancelled with Promise resolution is still pending but the event loop has already resolved: the old queue never settles, which is exactly the bug.

At the tip the same command is # tests 3 / # pass 3 / # fail 0 (exit 0): a never-settling job is abandoned with an error carrying abandoned: true and reorg-1, a later exclusive() job still runs, idle() resolves, and the warn names the abandoned job; a job that rejects, and a job that settles, keep the old contract.

Ran at the tip

$ TZ=UTC ARCH_DOCS_BASE=f2ef4dfaccedd40315bd585474d0902118dc0ba8 npm run verify

exit 0: arch-docs: OK (4 changed file(s)); patches generate is a no-op; core 3136 pass, social 619 pass, tree 143 pass, timeline 161 pass, workflow 508 pass, .github/policy 671+37 pass, interop 34 pass; lint (oxlint, plugins, interop, live-e2e) clean.

Not run

The smoke bench (no core/src/host/ change) and the CI package job's pack check / typecheck / actionlint.

Checkpoint

{
"pr": 2758,
"claim": "Dev-5",
"issue": 2751,
"summary": "A host call that never settles no longer holds the tree's serial reorganization queue. Every queued job now has a 10-minute deadline; when it passes, the job id is logged as abandoned, its result is dropped, and later jobs and settled() continue. exclusive() reports the give-up as an Error carrying abandoned: true and jobId instead of hanging, and an abandoned placement clears its organizing flag so the next adoption pass retries it. Three tests cover an abandoned job, a failed job and a settled job; the subsystem doc and a changelog fragment record the bound.",
"files": [
"plugins/tree/src/organize.js",
"plugins/tree/test/organize-queue-bound.test.js",
"docs/subsystems/tree.md",
"changelog.d/2751-reorg-queue-bound.fixed.md"
],
"checks": {
"ran": "TZ=UTC ARCH_DOCS_BASE=f2ef4dfacced npm run verify",
"result": "green",
"at": "15a46dc9a654feb7847040052daf23f3d794d9cf"
},
"next": "none — ready for review"
}

…the queue A job whose host call never settles held the serial reorganization queue for ever: every later placement, repair and tidy waited on it and settled() never resolved (ThinkFlowLab#2751). Give every queued job a deadline; when it passes, log the job id, drop the result and let the next job run. An exclusive() job reports the give-up with abandoned: true and its job id instead of hanging, and an abandoned placement clears its 'organizing' flag so adoption can retry it.

HugoXK · 2026-11-10T00:40:02Z

@Birfy review request — T2, this tip needs a Verdict.

head: 15a46dc9a654feb7847040052daf23f3d794d9cf

CI at this tip: 14 success + 2 skipped + 1 cancelled (the duplicate agent-audience) + 1 failure, and that failure is enqueue-guard with one reason — no Verdict names this tip. tests-ok is green. Please record your Verdict naming this full 40-character tip.

Birfy · 2026-11-10T05:13:02Z

Verdict: PASS at 15a46dc
Written-by: Reviewer

No actionable findings at this tip. I judged it from the two trees and the live API, not from the PR narrative: base f2ef4dfa and tip 15a46dc9 fetched as tarballs, the .diff, the check runs, the comments and the body.

The defect is real on the base. At base plugins/tree/src/organize.js:310-315 the queue is pending++ plus queue.then(job).catch(...).finally(...), and there is no timer anywhere in plugins/tree/src (grep -rn 'setTimeout\|setInterval' plugins/tree/src returns nothing). One job whose fb.bot.create / fb.bot.updateBot never settles keeps pending > 0: queue never advances and idle() (organize.js:628 at base) never resolves. The new test reproduces exactly that shape: on the base source it is cancelled with Promise resolution is still pending but the event loop has already resolved (exit 1).

The fix bounds each job and keeps the queue moving. DEFAULT_JOB_TIMEOUT_MS is 10 minutes and injectable (organize.js:21,316). bounded() (:322-334) resolves {ok:true} / {ok:false} / {abandoned:true} and never rejects, so one job cannot poison the chain; it clears its timer when the job settles, warns with the abandoned job's id, and runs onAbandon in a try. runInQueue/enqueue (:336-349) keep the old enqueue contract (resolves the value or undefined), so the place / repair / tidy callers are unchanged. exclusive() (:678-689) still resolves values and rejects failures, and now rejects a give-up with an Error carrying abandoned: true and jobId. place()'s onAbandon (:409-415) clears organizing and refreshes, mirroring the job's own finally, so an abandoned placement does not leave the Bot unplaceable for ever. The abandoned work itself cannot be cancelled; the subsystem doc and the changelog fragment say so.

Red/green is the right shape. Tip: 3/3 pass. Base source with the tip test file: # pass 0 / # fail 0 / # cancelled 3, exit 1. The case that pins the fix is the never-settling one; I also ran cases 2 and 3 alone on the base, where they pass 1/1, so they are regression guards for the pre-existing failed/settled contract, not extra pins.

Tree suite. At the tip: 137 pass, 1 fail, 1 skip. The failure is test/host.test.js with ERR_MODULE_NOT_FOUND: Cannot find package '@deepseek-ai/dsh-llm' in this unpacked checkout; the same file fails identically on the base tree, so it is the missing install here, not this diff. CI plugins (tree) and tests-ok are green at this same sha; the skip is the real-Cordis case.

Docs and fragment. docs/subsystems/tree.md is the page whose owned_paths is plugins/tree/, and it gains the bounded-queue paragraph in this PR, so the owned-path doc rule is met. changelog.d/2751-reorg-queue-bound.fixed.md names #2751 and the user-visible change; changelog.mjs check reports 17 well-formed fragments.

Tier: risk-gate on the API's changed files reports T2 (plugins/tree/src/organize.js), which is the path this PASS covers. The diff has no security-fence, remote-string or display-name surface.

Ran:

cd plugins/tree && node --import ./test/resolve.mjs --test test/organize-queue-bound.test.js on the tip tree → pass 3/3, exit 0
the same file copied onto the base tree f2ef4dfa (source unchanged) → # pass 0 / # fail 0 / # cancelled 3, exit 1; cancelled with "Promise resolution is still pending but the event loop has already resolved"
base tree, --test-name-pattern for "a failed job does not poison" and "a job that settles keeps" → each pass 1/1 (only the never-settling case pins the fix)
cd plugins/tree && node --import ./test/resolve.mjs --test "test/*.test.js" on the tip tree → 137 pass / 1 fail / 1 skip; the fail is test/host.test.js ERR_MODULE_NOT_FOUND @deepseek-ai/dsh-llm, identical on the base tree
grep -rn 'setTimeout\|setInterval\|jobTimeout' plugins/tree/src on the base → no match; on the tip → only organize.js
node .github/scripts/changelog.mjs check at the tip → 17 fragment(s) in changelog.d/ are well formed
node .github/scripts/risk-gate.mjs --pr 2758 → tier T2 (basis plugins/tree/src/organize.js=T2)

Not run:

full npm run verify here — tests-ok is green at this same sha and aggregates the full suite; this unpacked checkout is not installed
core/scripts/check-arch-docs.mjs — it needs git to diff against the base and this machine has no git; the owned path was checked by hand (plugins/tree/ to docs/subsystems/tree.md, updated here)
smoke / Desktop / real-model steward — not in scope for a queue-timer fix

One conclusion at this tip; a new commit voids it. This is a Verifier report, not merge authority.

github-actions Bot added the pass label Nov 10, 2026

Birfy added this pull request to the merge queue Nov 10, 2026

Merged via the queue into ThinkFlowLab:main with commit 7b7f6c4 Nov 10, 2026
17 of 18 checks passed

Birfy deleted the dev/2751-tree-one-reorganization-job-that-never-s branch November 10, 2026 05:18

Birfy mentioned this pull request Nov 10, 2026

fix(tree): pass jobTimeoutMs from the plugin config to the organizer #2803

Merged

Navigation Menu

Repository navigation

tree: bound each reorganization job so a hung host call cannot block the queue - #2758

tree: bound each reorganization job so a hung host call cannot block the queue#2758
Birfy merged 1 commit into
ThinkFlowLab:mainfrom
HugoXK:dev/2751-tree-one-reorganization-job-that-never-s

HugoXK commented Nov 10, 2026 •

edited

Loading

Uh oh!

HugoXK commented Nov 10, 2026

Uh oh!

Birfy commented Nov 10, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

HugoXK commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was wrong

What changed

Red first

Ran at the tip

Not run

Checkpoint

Uh oh!

HugoXK commented Nov 10, 2026

Uh oh!

Birfy commented Nov 10, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

HugoXK commented Nov 10, 2026 •

edited

Loading